Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhance: [2.4] Batch forward delete when using DirectForward (#37076) #37107

Merged
merged 1 commit into from
Oct 25, 2024
Merged

enhance: [2.4] Batch forward delete when using DirectForward (#37076) #37107

merged 1 commit into from
Oct 25, 2024

Conversation

congqixia
Copy link
Contributor

Cherry pick from master
pr: #37076
Related #36887

DirectFoward streaming delete will cause memory usage explode if the segments number was large. This PR add batching delete API and using it for direct forward implementation.

@congqixia
enhance: Batch forward delete when using DirectForward (milvus-io#37076)
1adfe46 
Relatedt milvus-io#36887

DirectFoward streaming delete will cause memory usage explode if the
segments number was large. This PR add batching delete API and using it
for direct forward implementation.

Signed-off-by: Congqi Xia <[email protected]>
@sre-ci-robot sre-ci-robot added approved size/XL Denotes a PR that changes 500-999 lines. area/internal-api labels Oct 24, 2024
@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Oct 24, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 24, 2024

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@congqixia
Copy link
Contributor Author

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 24, 2024

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@congqixia
Copy link
Contributor Author

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 24, 2024

@congqixia E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@congqixia
Copy link
Contributor Author

/run-cpu-e2e

@codecov Codecov
Copy link

codecov bot commented Oct 24, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 46.29630% with 116 lines in your changes missing coverage. Please review.

Project coverage is 80.56%. Comparing base (27a4fe0) to head (1adfe46).
Report is 5 commits behind head on 2.4.

Files with missing lines Patch % Lines
internal/querynodev2/services.go 0.00% 66 Missing ⚠️
internal/querynodev2/delegator/delta_forward.go 59.52% 27 Missing and 7 partials ⚠️
internal/querynodev2/segments/manager.go 0.00% 10 Missing ⚠️
internal/querynodev2/cluster/worker.go 89.74% 4 Missing ⚠️
internal/querynodev2/local_worker.go 0.00% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##              2.4   #37107      +/-   ##
==========================================
- Coverage   83.72%   80.56%   -3.16%     
==========================================
  Files         798     1066     +268     
  Lines      138606   165251   +26645     
==========================================
+ Hits       116045   133141   +17096     
- Misses      18220    27763    +9543     
- Partials     4341     4347       +6
Files with missing lines Coverage Δ
internal/distributed/querynode/client/client.go 93.02% <100.00%> (+0.30%) ⬆️
internal/distributed/querynode/service.go 83.25% <100.00%> (+0.15%) ⬆️
internal/util/mock/grpc_querynode_client.go 76.92% <100.00%> (+0.92%) ⬆️
internal/util/wrappers/qn_wrapper.go 82.35% <100.00%> (+0.53%) ⬆️
internal/querynodev2/local_worker.go 88.46% <0.00%> (-7.38%) ⬇️
internal/querynodev2/cluster/worker.go 84.56% <89.74%> (+1.64%) ⬆️
internal/querynodev2/segments/manager.go 76.66% <0.00%> (-1.45%) ⬇️
internal/querynodev2/delegator/delta_forward.go 63.50% <59.52%> (-0.50%) ⬇️
internal/querynodev2/services.go 86.63% <0.00%> (-4.75%) ⬇️

... and 297 files with indirect coverage changes

@yanliang567 yanliang567 added ci-passed manual-pass manually set pass before ci-passed labeled labels Oct 24, 2024
XuanYang-cn
XuanYang-cn approved these changes Oct 25, 2024
View reviewed changes
Copy link
Contributor

@XuanYang-cn XuanYang-cn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtg

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia, XuanYang-cn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@XuanYang-cn
Copy link
Contributor

/lgtm

@sre-ci-robot sre-ci-robot merged commit 6bc8aba into milvus-io:2.4 Oct 25, 2024
14 of 15 checks passed
@congqixia congqixia deleted the cp24/37076 branch October 29, 2024 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@XuanYang-cn XuanYang-cn XuanYang-cn approved these changes

@aoiasd aoiasd Awaiting requested review from aoiasd

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

Assignees

@XuanYang-cn XuanYang-cn

Labels
approved area/internal-api ci-passed dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement lgtm manual-pass manually set pass before ci-passed labeled size/XL Denotes a PR that changes 500-999 lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@congqixia @sre-ci-robot @XuanYang-cn @yanliang567